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SUBSTITUTE SPECIFICATION (Marked-Up) 

CONTROLLER BY THE DEVICES AND METHODS OF CONTROLLING MANIPULATION 
OF VIRTUAL OBJECTS ON A MULTI-CONTACT TACTILE SCREEN 

Related Application 

[0001] This is a $371 of International Application No, PCT/FR2005/000428. with an 
international filing date of February 23, 2005 (WO 2005/091104 A2, published September 29. 
2005), which is based on French Patent Application No. 04/50329, filed February 23. 2004. 

Technical Field 

[0002] The presen^nvention relates to th e ar e a of musical controllers . Th e pr e s e nt inv e ntion 
r e lat e s mor e A particularly to an man - machin e interface permitting, e.g., the control of music 
software or of a controller by a multi-contact tactile screen with the manipulation of virtual 
objects. 

Background 

[0003] The prior art alr e ady contains manual Manual -type software controllers are known . 
They ar einclude , e.g., potentiometers that can be manipulated by the user in the form of a 
console and control the different functions of music software. Such a console forms, e .g., th e 
subject matt e r of PCT application is disclosed in WO 01/69399. 

[0004] Th eOne disadvantage of this type of controller is that they are not very ergonomic for 
an efficient manipulation of software. Th e solution propos e d by th e pr e s e nt invention is One 
thought has been to implement a tactile screen for the manipulation of and the access to software 
functions. 



[0005] In the area of tactile controllers th e prior art alr e ady contains in PCT application^ WO 
03/041006 erand US 6,570,078 disclose musical controllers with tactile control on a matrix 
sensor. The technologies described there in th e s e docum e nts p ermit [[a ]]tactile control of the 
multi-contact type in which all the fingers can intervene for the control of software. 
[0006] However, th e se documonts those publications do not propos e contemplate a visual 
return of the manipulations since the different matrix sensors are of the opaque type. 
[0007] Th e prior art contains in Am e rican pat e nt application US 2002/005 1 08 (L e st e r Franck 
Ludwig) "Tactile, — Visual and Array Controllers for Real Time Control of Music Signal 
Processing, Mixing, Video and Lighting " discloses a system and a process for controlling in real 
time signal processors, synthesizers, musical instruments, MIDI processors, lights, video, and 
special effects during presentations, recordings or in compositional environments using images 
derived from tactile sensors, from matrices of pressure sensors, from matrices of optical 
transducers, from matrices of chemical sensors, matrices of body sensors and from digital 
processes. Th e inv e ntion of this Am e rican patent application That system furnishes touchpads, 
matrices of pressure sensors and matrices of body sensors as interfaces of tactile control, video 
cameras and matrices of light sensors such as optical transducers, matrices of chemical sensors 
and of other apparatuses for generating digital images from processes on computers or from 
digital simulations. The tactile transducers can be arranged on the keys of conventional 
instruments, be-attached to existing instruments or also be used to create new instruments or new 
controllers. The matrices of chemical sensors and the other apparatuses for generating digital 
images from computer processes or from digital simulations can be used to observe or simulate 
natural physical phenomena such as environmental conditions or self-organizing process 
behaviors. Matrices of scalars Scalar matrices or ef-vectors are processed in ord e r to extract 
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pattern limits, geometric properties of pixels within limits (geometric center, weighted moments, 
etc.) and information derived from a higher level (direction of rotation, segmented regions, 
pattern classification, syntax, grammars, sequences, etc.) that are used to create control signals to 
external video and visual equipment and for control or even algorithms. This invention lt also 
aBew sprovides MIDI and non-MIDI control signals to b e furnish e d . 

[0008] This American pat e nt application ^: does not propose contemplate a visual return of 
manipulations- — This Am e rican patent application and does not mention a command law. 
Finally, this pat e nt application! ! does not propos e contemplate technical solutionis]] to the 
masking phenomena that intervene when several figures are aligned or placed in an orthogonal 
manner on the sensor. The resolution of these problems is indispensable for realizing a multi- 
contact tactile sensor. 

[0010] Th e prior art also contain s in Am e rican pat e nt US 5,027,689 (Yamaha) "Musical 
Tone Generating Apparatus " discloses an apparatus for generating musical sounds. TfeisThat 
apparatus comprises a device for generating positional information for generating information 
about the position of musical instruments (PS) as values of plane coordinates. This information 
(PS) is stored in a memory device or determined in a selective manner by a manual operation. 
The apparatus also comprises a device for the conversion of information for converting the 
information (PS) into information for controlling parameters of musical sounds (PD). This PD 
control information controls the source signals of musical sounds (SI 1, S12 and SI 3) for 
generating a sound field corresponding to the position of musical instruments arranged on a 
stage. This allows an operator to verify the positions of musical instruments on a stage, thus 
supplying the sensation of being in a true live performance. 
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[0011] This American pat e nt lt mentions a multi-contact a but it is only two contacts on an 
axis and not in Cartesian coordinates. The apparatus of this American patent only functions 
linearly for the multipoint option and does not allow tracking (following of trajectory). 
Moreover, the apparatus of this Am e rican patient requires a plurality of sensors specific to each 
of the instruments wh e reas th e pr e sent inv e ntion r e lat e s to a g e n e ric s e nsor . 
[0012] Th e prior art or also contains US 5,559,301 discloses a solution of the musical 
controller type in the form of a tactile screen with visual return of the manipulated objects-by-US 
pat e nt 5, 559, 301 . However, this pat e nt it describes predefined objects (essentially of the sliders 
type and circular potentiometer type). These object types are falheF-limiting and can prove to be 
not very ergonomic for special manipulations. Moreover, the acquisition mode described in this 
pat e nt is not in real time. In fact, an icon must first be activated by a first contact with a finger, 
then the manipulated object, and the values are only updated after the icon has been released. 
This solution That system does not allow [[a ]]management in real time of the parameters 
associated with the object. Finally, the tactile sensor u se d in this pat e nt is a "mono-contact" 
sensor that permits the acquisition, e.g., only for a single finger and therefore the control of a 
single object at a time. This characteristic is very limiting for an efficient manipulation of 
objects. 

Summary 

[0013] This invention relates to a process for controlling computerized equipment with a 
device including a multi-contact bidimensional sensor that acquires tactile information and a 
calculator that generates command signals as a function of the tactile information, including 
generating graphical objects on a screen placed under a transparent multi-contact tactile sensor, 
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each graphical object associated with at least one specific processing rule such that the sensor 
delivers during each acquisition phase a plurality of tactile information, and each piece of the 
tactile information forms an object of a specific processing determined by its localization relative 
to a position of one of the graphical ob jects. 

[0014] This invention also relates to a device for controlling computerized equipment 
including a multi-contact bidimensional sensor for acquisition of tactile information, a viewing 
screen arranged under the bidimensional tactile sensor, a memory for recording graphical objects 
that are each associated with at least one processing rule, and a local calculator that analyzes 
positions of acquired tactile information and applies a processing rule as a function of the 
position relative to the position of the graphical objects. 

Brief Description of the Drawings 

[0015] The disclosure will be better understood with the aid of the following description 
given below solely by way of explanation of a selected, representative example with reference 
made to the attached figures in which: 

Fig. 1A is a functional diagram of a controller; 

Fig. IB represents the structure of the controller associated with the functional diagram; 

Fig. 1C represents the functional diagram of the different stages of the processes for the 
acquisition of data coming from the sensor, of the creation of cursors associated with the 
different fingers, of the interaction with the graphical objects and of the generation of control 
messages; 

Fig. 2A is a description of the tactile matrix sensor; 
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Fig. 2B describes the first stage of the scanning functioning of the sensor in order to 
obtain the multi-contact information; 

Figs. 2C 2E and 2F explain the resolution of problems of orthogonality; 
Fig. 2D is a functional diagram of the capture interface; 

Figs. 3A to 3F are diagrams explaining the stages for the creation of cursors, filtering, 
calculation of barycenter, mapping and of the control of graphical objects; 
Figs. 4 and 5 represent different examples of graphical objects; 

Figs. 6 to 10 represent different examples of combinations of graphical objects on the 
controller; and 

Fig. 1 1 illustrates the network use of the controller associated with the computer of the 

user. 

Detailed Description 

[0016] In all th e following th e The term "multi-contact" defines a tactile sensor that allows 
the-acquisition of contact zones of several fingers at a time in contrast to "mono-contact" sensors 
that only allow the-acquisition for a single finger or for a stylus as, e.g., in th e pr e c e ding patent 
US 5, 559,301. 

[0017] Th e pr e s e nt inv e ntion has the probl e m of r e ctifying th e disadvantages of the prior art 
by proposin g We provide a screen for multi-contact tactile musical control with visual return of 
the different actions of the user on parameterable objects. 

[0018] In order to do this th e pr e s e nt invention is of tho typ e d e scrib e d above and is 
r e markabl e in it s broad e st m e aning in that it conc e rns We also provide a process for th e control 
e fcontrolling computerized equipment b ywith a device comprising a multi-contact bidimensional 
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sensor for the acquisition of tactile information as well as compris e s calculating means 
generating command signals as a function of this tactile information, charact e rized in that it 
compris e s and a stage for th e g e n e ration o f generating graphical objects on a screen placed under 
a transparent multi-contact tactile sensor, each of which graphical objects is associated with at 
least one specific processing law, tha twherein the sensor delivers during each acquisition phase a 
plurality of tactile information, and that-each piece of thisthe tactile information forms the object 
of a specific processing determined by its localization relative to the position of one of these 
graphical objects. 

[0019] The proc e ssings pr e f e rabl v process steps may comprise a bounding zone detection of 
the contact zone of an object with the tactile sensor. 

[0020] The proc e ssings advantag e ousl v process may also comprise a barycenter detection. 

[0021] It pr e f e rablv may further comprise[[s]] stages for the-refreshing ef-graphical objects as 

a function of the processings carried out during at least one previous acquisition stage. 

[0022] According to an e mbodim e nt it The process may comprise[[s]] a stage for editing 

graphical objects consisting i ncluding generating a graphical representation from a library of 

graphical components and functions and in-determining an associated processing law. 

[0023] The acquisition frequency of the tactile information is pr e f e rabl v may be greater than 

50 Hz. 

[0024] The pr e s e nt inv e ntion We also conc e rns provide a device for th e control o f controlling 
a computerized piece of equipment comprising a multi-contact bidimensional sensor for the 
acquisition o f acquiring tactile information, charact e riz e d in that it furth e rmor e compris e s a 
viewing screen arranged under the bidimensional tactile sensor, as w e ll as a memory for 
recording graphical objects that are each associated with at least one processing law, and a local 



7 



calculator for analyzing the position of acquired tactile information and the-application of a 
processing law as a function of thisthe position relative to the position of the graphical objects. 
[0025] Mor e ov e r, it is pref e rabl v The device may be connected to a hub (multi-socket 
network) for formin gt o form a network of controllers. 

[0026] This multi-contact bidimensional tactile sensor is advantageously a resistive tile. 
[0027] Furthermore, thisthe device pr e f e rabl vm ay comprise[[s]] a network output suitable 
for receiving a network cable. 

[002 8 ] Th e inv e ntion will b e bett e r und e rstood with th e aid of th e following d e scription 

giv e n b e low sol e ly by way of e xplanation of an e mbodim e nt of th e inv e ntion with r e f e r e nc e 
mad e to th e attach e d figur e s in which: 

Figure 1 A is a functional diagram of the controll e r in accordanc e with th e inv e ntion. 

Figur e IB repr e s e nts th e structur e of the controll e r a s sociat e d with th e functional diagram 
of the inv e ntion. 

Figur e 1C r e pr e s e nts th e functional diagram of th e diff e r e nt stag e s of the proc e ss e s for 
th e acquisition of data coming from the s e nsor, of th e cr e ation of cursors associated with th e 
different fingers, of th e int e raction with the graphical obj e cts and of th e g e n e ration of control 
m e s s ag e s. 

Figur e 2A is a d e scription of the tactil e matrix s e nsor. 

Figur e 2B d e scrib e s th e first stage of th e scanning functioning of th e sensor in order to 
obtain the multi contact information. 

Figur e s 2C, 2E and 2F e xplain th e r e solution of probl e ms of orthogonality. 
Figur e 2D is a functional diagram of th e captur e int e rfac e . 
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Th e seri e s of figures 3A to 3F e xplain the stages for th e cr e ation of cursors, filt e ring, 
calculation of baryoent e r, mapping and of th e control of graphical obj e cts. 
Figur e s 4 , 5 repr e s e nt diff e rent e xampl e s of graphical obj e cts. 

Figur e s 6 to 10 r e pr e s e nt diff e r e nt e xampl e s of combinations of graphical obj e cts on th e 
controll e r. 

Figur e 1 1 illustrates th e n e twork us e of th e controll e r associat e d with th e computer of th e 

[0029] In all of the following description, the control is performed on a computerized piece 
of equipment that can be, e.g., a music software, a controller, audiovisual equipment or 
multimedia equipment. 

[0030] As flgufe sFigs. 1 A, IB and mor e pr e cis e ly 2A illustrate, the first basic element of th e 
pr e s e nt inv e ntion is the matrix sensor 101 necessary for the— acquisition (multi-contact 
manipulations) with the aid of a capture interface 102. The sensor ean lOl may be divided, if 
necessary, into several parts in ord e r to accelerate the-capture, with each part being scanned 
simultaneously. 

[0031] The general principle is to create as many cursors (such as a mouse cursor) as there 
are zones detected on the sensor and to follow their developments in time. 
[0032] When the user removes hts the user's fingers from the sensor^ the associated cursors 
are destroyed. 

[0033] In this manner^ the position and the-development of several fingers are captured 
simultaneously on the sensor. This is a multi-contact capture that is quite innovative for this type 
of controller. 
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[0034] The sensor us e d for th e embodim e nt of the invention is may be a resistive tactile 
matrix tile of a known type. 

[0035] Resistive tactile matrix tiles are composed of 2 superposed faces on which tracks of 
ITO (indium tin oxide), that is a translucent conductive material, are organized. The tracks are 
laid out in lines on the upper layer and in columns on the lower laye r, formin g and a matrix (efe 
Figur eas shown in Fig. 2A). 

[0036] The two conductive layers are insulated from one another by spacing braces. The 
intersection of the line with the column forms a contact point. When a finger is placed on the 
tile a a column or columns situated on the upper layer are put in contact with a line or line situated 
on the lower layer, thus creating one or several contact points (of. Figure as shown in Fig. 2B). 
[0037] A variant of th e inv e ntion advantag e ously consists in r e placin g lt is possible to replace 
the braces by a transparent resistive material (e.g., a conductive polymer) whose resistance varies 
as a function of the pressure, which resistance drops if a sufficient pressure force is exerted. In 
this manner^ it would is also be-possible to extract the pressure exerted on the surface by 
performing a resistance measurement at each line-column intersection. 

[0038] As concerns the musical or audiovisual use of these tiles, it is imperative to measure 
the activity of a finger with a maximum latency of 20 ms. 

[0039] The state of the tile is measured at least 100 times per second, which tile can be 
divided into several zones in ord e r to pefefm perform a parallel processing on these zones. 
[0040] Thus, according to th e inv e ntion the sampling frequency of the tile is may be at least 
100 Hz. 

[0041] Another basic element is the electronic device for scanning the tactile tile that allows 
the simultaneous detection of several contact points on the matrix sensor. In fact, the known 
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methods of acquisitions for this type of sensor[[s]] do not allow the detection of several 
simultaneous contact points. 

[0042] The known methods known in th e past do not allow the problems illustrated by 
figweinFig. 2C to be solved. 

[0043] If a simultaneous measurement of all the lines is performed while feeding a column, 
problems of orthogonality arise. Contact point No. 1 will mask contact point No. 2. Likewise, if 
a line is measured when all the columns are fed, contact point No. 2 is masked by contact point 
No. 1. The solution propos e d for solving this problem consistsi g in performing a sequential 
scanning of the sensor. 

[0044] The columns are fed, e.g., at 5V in turn and the level of the lines (high or low level) 
measured sequentially. 

[0045] When one of the columns is placed under voltage* the others are in high impedance m 
ord e r to prevent the propagation of current into the latter. 

[0046] Thus, column 1 is fed at first while the other columns are in high impedance. 
[0047] The lines are measured sequentially, that is, one after the other. The value on the first 
line is read initially while all the other lines are connected to ground. Then, line 1 is connected 
to ground and the value on line 2 is read and so forth until the value of all the lines has been read. 
[0048] Column 1 then passes into the high impedance state and column 2 is fed. The reading 
of the state of each of the lines recommences. 

[0049] The scanning is performed in this manner up to the last column. 
[0050] As the goal is to form a multi-contact tile, the total scanning of the matrix is carried 
out at an elevated frequency in ord e r to obtain the value of each of the intersection points of the 
tile several times per second. 
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[0051] The device permitting the acquisition of the tile data is illustrated in figureFig. 2D, 
representing the algorithm of the acquisition of a tile comprising 100 lines (L) and 135 columns 
(C). 

[0052] Certain problems in the masking of a point by one or several other points can appear. 
[0053] In fact, the resistance of the transparent material (ITO) composing the columns and 
the lines increases proportionately to the length of the tracks. Thus, the potential measured at the 
lower left corner of the sensor will be greater than the potential measured at the upper right 
corner. 

[0054] In figuresFigs. 2E and 2F, the cloud of points absorbs a large part of the electrical 
potential of the fed column. The potential measured at the isolated point is therefore too low to 
be detected. 

[0055] TheA solution to this problem consists ^ in using a voltage comparator piloted 
digitally at the output of the line in ord e r to determine whether the tension observed is sufficient 
for being considered as resulting from the action of a finger on the tactile tile. The reference 
value of the comparator (comparison threshold) is decremented at each line measure. Thus, the 
comparison values of the last lines are lower than those of the first lines, which allows the 
contact point located at the lower left or the upper right to be detected in the same manner. 
[0056] Thus, e.g., thea complete sampling of the tile is performed at least 100 times per 
second for the columns and the lines. 

[0057] The data from capture interface 102 thus form an image representative of the totality 
of the sensor. This image is placed in memory so that a program can proceed to the filtering, the 
detection of the fingers and to the creation of the cursors . R e f e r for this offoct to figur e as seen 
in Fig. 1. 
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[0058] The filtering phase illustrated by figweFig. 3B consists in e liminating th e eliminates 
noise that might be generated by the acquisition interface or the sensor itself. It is considered 
that only the clouds of several contact points can correspond to the pressure of a finger. 
Therefore, a bounding zone detection is carried out in ord e r to eliminate isolated contact points. 
[0059] The following stage consists in associatin g associates a cursor with each support point 
(figw eFig. 3C). To this end A the barycenter of each bounding zone is calculated. When a finger 
is released^ the corresponding cursor is freed. 

[0060] The program executed locally by the main processor allows these cursors to be 
associated with graphical objects that are displayed on screen 105 in order to manipulate them. 
At the same time^ the local program uses these cursors for generating control messages addressed 
to the host computer or the controlled apparatus. 

[0061] Furthermore, the program comprises a simulator of the physical models allowing the 
modification of the interaction laws between the cursors and the graphical objects. Different 
physical models can be employed: spring-loaded system, vibration of a string, management of 
collisions, the law of gravity, electromagnetic field and the like . 

[0062] The program considers the positioning of the cursors and on which graphical object 
each is located. A specific processing is supplied to the data coming from the sensor as a 
function of the object considered. For example, a pressure measurement (corresponding to a 
development of the spot made by the finger on the tactile tile in a short interval of time) can be 
interpreted. Other parameters can be deduced as a function of the nature of the object: the 
acceleration, speed, trajectories, etc. Algorithms of recognition of form can also be applied m 
ord e r to differentiate different fingers. 
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[0063] The main program 103 also transmits the data to be displayed on screen 105 to 
graphical interface 104. Moreover, this graphical interface is constituted byof a graphical 
processor. This graphical processor is, e.g., of a known type. The latter can be constituted byof 
primitive graphical functions allowing, e.g., the displaying of bitmap, ef-fonts of polygons and ef 
figures in 2 and 3 dimensions, the-vectorial design, the-antialiasing, the-texture mapping, the 
transparency and the-interpolation of colors. 

[0064] In this pr e s e ntation of th e inv e ntion th eThe main program may also comprise[[s]] an 
analyzer of mathematical expressions that allows mathematical functions to be inputted and 
calculated in real time. These functions allow the values of any variable to be modified. For 
example, the coordinates (x, y) of a cursor inside an object can be considered as two variables 
comprised between 0 and 1. The expression analyzer allows an expression of the type 
"x* 1000+600" to be created in order to obtain a new variable whose value is comprised between 
600 and 1600. The variable obtained allows the control, e.g., of the frequency of an oscillator 
comprised between 600 and 1600 hertz. 

[0065] The mathematical expressions can be applied to scalar values as well as to vectors. 
[0066] The expression analyzer is a tool that allows real-time calculations to be performed 
on the variables of objects. 

[0067] Local program 103 also performs a formatting of data in the form of messages for 
network port 106, that communicates it to the computer on which the computer applications are 
performed. 

[0068] The network interface is, e.g., an Ethernet 10/100 baseT standard interface that 
communicates by packets with the protocol TCP/IP. It can also be a network interface of the 
wireless type. 
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[0069] It should be noted as illustrated in figufeFig. 1 1 that the Ethernet connection offers 
the user the possibility, by using a simple hub (multi-socket network), of indefinitely expanding 
histhe control apparatus by constituting a network of controllers in accordanc e with th e 
inv e ntion , 

[0070] The controller or controllers present in the network then communicate among 
themselves and with the host computer in the form of the reciprocal sending of messages. 
[0071] Furthermore, the unit constituting the machine is fed by a battery (not shown) of a 
known type or by an AC adapter. 

[0072] Finally, an interface editor 107 at the level of the computer of the user allows the 
interface, that is, the totality of the graphic objects displayed on screen 105, to be programmed in 
a graphical manner. In this e mbodim e nt of th e invention th e The interfaces are mav themselves 
be organized in scenes, that are higher hierarchical structures. In fact, each scene comprises 
several interfaces. The user can interchange the interfaces with the aid of a button keyboard or a 
control pedal board connected to input-output port 109. 

[0073] Another function of the interface editor is to assign the control data to the parameters 
that the user wishes to control. 

[0074] The user has at hi sthe user's disposal, e.g., a library of parameterable graphical 
objects allowing the composition of different interfaces according to the application desired. 
FigwesFigs. 4 and 5 represent different graphical objects placed at the disposition of the user. 
[0075] They can be predefined and dedicated quite particularly to music or to the control of 
audiovisual equipment or computerized apparatuses. For example, a linear potentiometer 403, 
404 is particularly adapted to control continuous parameters such as the volume of a sound 
signal, the frequency of a filter. A serrated wheel 401 can serve, e.g., to control the playing of an 
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audio or video reader. The objects can also be freely developed with a development kit (SDK) of 
a known type 109. The development kit furnishes access to the primitive graphical functions of 
the controller. 

[0076] Interface editor 107 thus allows the user to readily create personalized control 
interfaces. It is a software executed on the user's computer. It is composed of a main window 
representing the tactile surface of the tile on which graphical objects from a library of proposed 
objects can be placed. The manipulation and placing of objects on the surface are performed, 
e.g., with the mouse. The object placed on the window is displayed at the same time on the 
controller and the object is recorded in a memory of the controller. It can subsequently move or 
re-dimension the objects at its convenience. 

[0077] In addition to the positioning of graphical objects on the main window, other 
secondary windows allow the regulation of different parameters inherent in the objects (graphical 
properties, physical behavior). For example, a button 402 can also act as a switch or as a trigger. 
In the case of the trigger mode A a pressure measurement can optionally be performed. Another 
example of a parameterable object is area 2D (503, 544) of which the principle consists in cludes 
moving pawns inside a delimited zone. The number of pawns present in area 2D is a 
parameterable option. The area can be configured in uniplan mode, a mode in which the pawns 
enter into collision with each other, or multi-plan, a mode in which the pawns are placed on 
distinct superposed planes. Physical parameters can also be configured: the coefficient of 
friction of the pawns on the plane, the rebound and the attraction of the pawns on the edges and 
among themselves. 

[0078] The editor also permits the objects present on the surface to be listed and the creation 
of functions and of variables with the expression analyzer. 
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[0079] Thus, the objects have by default a certain number of variables (x, y, z...) 
corresponding to their primitive axes. These variables are always comprised between 0 and 1 
and vary in the form of 32-bit numbers with floating comma. The user must be able to "connect" 
these variables to other values more representative of what he desires to control. Thus, the 
expression analyzer furnishes the possibility of creating new variables with the aid of simple 
mathematical expressions. For example, a rectilinear potentiometer has a primitive axis that is x. 
If the user wishes to control the frequency of 500 to 2500 Hz he must create a variable 
a=2000x+500. 

[0080] Status display options are also desired. They permit a visual control of the state of a 
parameter. 

[0081] The further treatments to be applied to the objects at the level of the main calculating 
unit 103 by the manipulation on the tile are specific to each type of object. 
[0082] In fact, a circular movement of the finger on a virtual linear potentiometer (403, 404) 
should not have an effect on the state of the potentiometer whereas it should modify the state in 
the case of a circular potentiometer 401. Likewise, certain objects can only take into account a 
single finger (the linear potentiometer, for example) at a time whereas others can accept the 
interaction of several fingers (keyboard, area 2D). 

[0083] For example, the "area 2D" (503, 504) is a rectangular surface containing a certain 
number of pawns, each with its own position. The pawns can be moved by the user. 
[0084] The principle is to put in place a physical system for the totality of the objects, that is, 
e.g., that the pawns moved by the user acquire a speed of inertia that they retain when the user 
lets them go; the pawns subjected in this manner to their own speed will rebound on the edges of 
"area 2D" and also rebound among themselves. Furthermore, they will be subjected to forces of 
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attraction/repulsion on the edges and on the other pawns as well as to a coefficient of friction on 
the surface of area 2D for stopping the pawns at the end of a certain time. All these parameters 
will be parameterable. 

[0085] Another variant of area 2D consists in cludes applying a physical law of the "spring- 
loaded" type. A virtual rubber band is stretched between each cursor and each pawn. The user 
can modify the behavior of this object by configuring the friction and the interpolation factor. 
These properties can also be modified in real time with the aid of other objects. 
[0086] Another example is the "Multislider" 501, a table of cursors whose numbers can be 
configured. The typical use is the controlling of a graphic equalizer or of a spectral envelope. 
The difference between a "multislider" and several simple juxtaposed linear potentiometers is 
that the totality of the cursors can be modified in a single touch by sliding the finger. The 
multislider can also be used as a discrete string. For this, it is sufficient to apply to it the physical 
model of a string whose tension is parameterable by the user. 

[0087] A visualization of different examples of interfaces uniting different types of objects is 
illustrated by figur e s Figs. 6 to 9, in which several objects described above can be observed. 
[0088] Figure. 6 shows an arrangement of 6 areas 2D (601) containing 1 pawn each. This 
interface could control, e.g., six different filters assigned to one or several sound sources. In this 
instance^ the abscissa movement of each pawn in each zone controls the frequency of the filter 
whereas the ordinate movement controls the quality factor or the width of the filter band. 
[0089] Figwe. 7 shows an example of the control of a synthesizer or of a sampler of a known 
type. The interface is composed by a tempered keyboard 704 controlling the pitch of the sounds, 
by a group of four vertical potentiometers 703 allowing the control, e.g., of its dynamic envelope 
(attack time, hold level, release time). An area 2D (701) containing 3 pawns allows the control, 
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e.g., of effects applied to the sound (reverberation, echo, filters). A matrix of 16 buttons 792 can, 
e.g., release 16 different recorded musical sequences or also call up 16 prerecorded 
configurations of the previously described controls. 

[0090] Another example of an application of th e inv e ntion is illustrated by figur e Fig. 8 
showing the control of a device for the broadcasting of different sound sources into space on a 
device constituted by several loudspeakers. In this configuration an area 2D (801) representing 
the broadcasting space contains 4 pawns 801 corresponding to four sound sources. Area 2D also 
contains 5 icons 802 representing the position of five loudspeakers. The level and/or the phase 
of each sound source relative to each enclosed space is regulated by moving the different pawns 
802, which determines its emplacement in the space. Moreover, a group of four linear 
potentiometers 803 allows the relative level of each source to be regulated. A unit of four 
buttons 804 allows each sound source to be activated or deactivated. 

[0091] Another example is illustrated by figur e in Fig. 9 that shows the control of a 
synthesizer or a sound generator according to a configuration different thanfrom that shown by 
figure in Fig. 7. Here, the frequency of the sound generator is controlled by four virtual strings 
903. The initial tension (the pitch) of each string can itself be controlled, e.g., by a linear 
potentiometer 902. An area 2D 10, e.g., control other parameters of the sound generator such as 
the output level, the sound quality, the panning, etc. 

[0092] Figure. 10 shows the control of equipment for audio and/or video editing of a known 
type. A serrated wheel 1001 allows the rate of reading the audio and/or video sources to be 
controlled. Status display object 1002 allows the positioning of the reading to be represented 
according to a format (hour, minute, second, image) of a known type. A set of buttons 1003 
allows access to the functions of reading and editing of the controlled apparatus. 
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[0093] The inv e ntion was devices and methods described above are by way of example. It is 
understood that an e xp e rt one skilled in the art is capable of realizing different variants of the 
inv e ntion devices and methods without departing from the scope of the paten tappended claims . 
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